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ABSTRACT 

Intended as an aid to school systems in planning and 
executing evaluation of educational projects and programs, this 
handbook presents the technical aspects of evaluation in a practical 
way. The chapters of the handbook are: 1. Assessment of Needs; 2. 
Establishment of Objectives; 3. Measurement of Objectives; 4. 
Dissemination of Evaluation; and 5. Critique of an Evaluation Report. 
A bibliography on the evaluation of educational programs is provided. 
In the appendix to the handbook, the Critique of Evaluation Report is 
given. (DB) 
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PREFACE 



Essential to every educational project is the process of appraising the results 
of the expenditure effort and money. Continuing evaluation enables the 
project director and staff to consider what's working well, how expectations are 
being met and what modifications are indicated. 

In short, evaluation not only "finds out" but it also involves the next step, 
that of assuming responsibility for ''doing something" about the fmdings. 

It is the aim of this handbook to help make evaluation sound and appro- 
priate. The intent is to offer practical assistance to school systems in planning 
and executing evaluation of educational projects and programs. 

The intent is to present the technical aspects adequately, yet in a practical 
and, hopefully, interesting way. The authors do not presume that this guide 
obviates a project director*s familiarity with more comprehensive treatments 
of evaluation theory and practice. Therefore, we call the reader*s attention to 
the bibliography at the end of this guide. 

We wish to acknowledge a number of individuals who contributed to 
final product. Much of the material herein was presented originally by the fol- 
lowing conference leaders m a five-session Title III evaluation seminar held in 
January— February 1970: Ann Marie Bemazza, George Bondra, Ronald 
Cote'. Muriel Gerhard, Toe Gordon, Robert Hale, James Mathews, Robert 
Neanne, David Shafer. In addition, we are indebted to Dr. Robert Stoughton, 
Associate Commissioner — Instructional and Pupil Services, Dr. William 
Nolan, Chief— Bureau of Elementary and Secondary Education, Dr. James 
Burke, Evaluation Consultant, and Mr. Wallace Roby, Title I Consultant, 
for their thoughtful contributions to the content of this guide. 

Finally, we thank Ann Hedgcock for her imaginative approach to the illus- 
trations. 

We are pleaded to share credit with the above individuals for whatever merit 
this guide has. 



George D. Kinkade 
Roger E. Richards 
March 1972 
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CHAPTER 1 ASSESSMENT OF NEEDS 



Increasingly the concept of critical educational needs is stressed in connection 
with program development and evaluation. Project applicants are expected to 
design proposals to meet these needs. The National Needs Assessment study 
and State assessment activities may provide clues for project direction. How- 
ever, local assessment of need remains a must for the development of worthy 
projects. 

State and Federal programs that encourage the project approach have many 
criteria in common. Evidence is usually requested that the project demonstrate 
solutions to identified critical needs and substantially increase the educational 
opportunities for children in the area. 

Needs assessment attempts to Hnd those educational objectives which need 
to be accomplished in a given instructional situation. A desired learner outcome 
is identified and the learners' current status in respect to that outcome is ascer- 
tained. The difference between current status and the desired status is consid- 
ered to be an educational need. 



desired 
learner 
outcomes 



mmus 



current 
learner 
status 



equals 



an 

educational 
need 



Many projects may have come out of the creative genius of individuals who 
said "this is needed". There should be room for the implementation of such 
dreams. More and more however requirements call for persons broadly repre- 
sentative of the cultural and educational resources of the community to par- 
ticipate in all appropriate aspects of planning and conducting project activities 
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There are in every school system people who are not benefitting from the 
educational process. Their needs are often not considered or even known. The 
start should be a systematic assessment of the community, the educational and 
the community at large. Once having identified a number of education needs 
they should be ranked in some way so that the educational system can be di- 
rected toward the most important ones. Here again is an opportunity to involve 
the community, fellow educators and above ail the learners themselves. This 
involvement will pay off in the long run. The project will be better understood 
and accepted. 




There are other ways to determine educational needs and these must also be 
explored. One way might be called "divine revelation". The people giving out 
the money reveal needs and priorities. The government sources, the State Dept. 
of Education, the regional planning agencies and foundations are all big broth- 
ers in this process. 
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''Bring in theexperts"may be another technique for determining needs. They 
may be outside experts from the University. They may also be the practitioner, 
the person in the school system who has to do the job. 

Science may be able to help— statistics, measurement, hard data that can be 
interpreted. Information may be available concerning similar programs, 
relevant research findings and views of recognized experts. A pilot program or 
study may help determine whether or not the needs can be met and if they are 
really there. 

Needs assessment attempts to specify the kind of terminal behavior ultimately 
desired. What ought to be present that is not present. Needs assessment iden- 
tifies what is happening now— if it is good it can be left that way or enhanced— 
if it is not good, plan tochange it to something better. 
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CHAPTER 2 ESTABLISHMENT OF OBJECTIVES 



One of the shortcomings of many project evaluations is the absence of ob- 
jectives related to changes in students. The objectives evaluated are often state- 
ments of means rather than ends, or process rather than products. There may 
well t*c a need for evaluation of the process but this should be secondary in 
importance to the product objectives. 

Each project objective should serve to suggest one or more descriptions of 
ways in which students can be observed to have attained that objective to some 
degree. 



Project Objective 
The improvement of readiness of 
Kindergarten pupils for formal 
instruction 



Desired Learning Behavior 

1. Gain skills in using words and rec- 
ognizing names 

2. Gain skill in number size and quan- 
tity relationship 

3. Gain skill in identifying and matching 
colors 



If the desired learning can be stated as observable behavior, this is best from 
the evaluation viewpoint. 

Behavioral objectives mean in effect that if learning takes place, some kind 
of behavioral change must be observed. Behaviorally "the student runs** or 
"the student writes'* or "the student yelk*\ The student thinks, however, is con- 
sidered to be non-behavioral and as such presents another problem. 



When dealing with words like "thinking" or "appreciation'', we can*t directly 
measure or even know when it is taking place. Our only criteria can be what the 
student demonstrates. He frequently discusses music; he borrows records; he 
participates in music activities. Maybe those things are an inference toward 
appreciation. 

A problem in semantics arises when people attempt to describe behavior. 
Some clarification may be obtained by dividing behavior into three domains. 

The cognitive domain emphasizes the mental processes beginning with the 
concrete acquiring of knowledge and continues through the more abstract 
processes of analysis and synthesis. 




"Johnny will display knowledge of the U.S. Constitution as measured by a 
teacher— made test (75% correct) 



Theaj(/^c//vedomain behaviors are harder to state or evaluate. The emphasis 
is on the emotional processes, beginning with the simple behaviors to the com- 
plex process of total characterization. 



"Johnny will respond positively to an inventory designed to assess student 
attitude toward symphony music. (60% positive) 

The psychomotor variable describes those behaviors which involve neuro- 
muscular coordination. Many of these activities ought to be easily observable 
but in the past have been virtually ignored by educational evaluators. Hand- 
writing and physical education experts have been some use of this domain but 
the possibilities are much broader. 




seconds'', (teacher judgment) 



These are by no means clear-<;ut domains, they intertwine. No behavior is 
totally in one area and most behaviors involve all areas, (see working paper **1) 

Stating Educational Objectives in nonbehavioral terms is relatively easy. 
However, the result may be something as ambiguous as: "This course will 
cover ..." or, "the instructor mil. . . " First attempts to state objectives in 
terms of student behavior may read: ''students will develop appreciations . . . 
or "students will understand . . If the teacher is too determined he will state 
that: "the student will really understand.. . " The problem is what behavior 
shall we look for in evaluating students' "understanding" and "appreciation". 

Objectives stated in unambiguous terms are essential to individualized learn- 
ing. Individualized learning has caused educators to become aware of the neces- 
sity of thinking specifically, and speaking precisely when stiting the objectives 
of any educational endeavor. Only then is it possible to ascertain when the ob- 
jectives have been reached by the learner. 



1. IDENTIFY 



—select, choose, differentiate, point, 
pick up, place 

—arrange, (in specified sequence of 
pattern) 

—designate, list 

—show 

—name a different number of charac- 
teristics; portray 
—make, build, draw, illustrate 
—name, formula (formulate) 
— calculate 
—explain, defme 

Remember we want objectives that are specific, behavioral, and above all 

measurable! 



2. ORDER 

3. NAME 

4. DEMONSTRATE 

5. DESCRIBE 

6. CONSTRUCT 

7. STATE RULE 

8. APPLY RULE 

9. INTERPRET 



After an objective has been formulated as clearly as possible and its impor- 
tance confir.ned by appropriate judges then measures must be devised to assess 
the attainment of the objective. Norm referenced tests which aim to discriminate 
among individuals may or may not be suitable to detect changes brought about 
by an educational treatment. Criterion referenced tests on the other hand are 
designed specifically for use in detecting the siaf.is of an individual or a group 
in regard to a specific objective, and may better serve the purpose. However, to 
use this approach to measurement we must set criterions of acceptable perfor- 
mance (working paper HI) 

/ 
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WORKING PAPER I 



The Behavioral Domains 
Both overt and covert behaviors may be classified into three domains: 
Cognitive —deals primarily with mental or intellectual processes— 

the processes of thinking, of knowing, and of problem- 
solving. 

Affective V —deals with feelings, emotions, attitudes, values, in- 
terests and appreciations. 

Psychomotor —encompasses manual, motor, neuromuscular or phys- 
ical skills. 

Based on the above definitions, review the following behaviors and categorize 
each by placing a C in front of those which are highly illustrative of the cognitive 
domain, an A in front of those which are of the affective domain and a in 
front of those which are psychomotor. 

1. The pupil demonstrates a more positive self-concept as measured 

by the How I Like Myself Inventory. 

2. The teacher enrolled in a teaching methods course develops a 

value for a certain teaching technique as measured by his ability, 
without being asked to discuss the factors which make it a good 
method for him. 

3. Given a variety of alternatives in selecting a college, the pupil 

makes his selection and states the criteria which he used in reach- 
ing his decision. 

4. f he pupil strongly defends his viewpoints as measured by teacher 

observation. 

5. The pupil demonstrates his competency in the Smith Method of 

Exercising by doing push-ups at a frequency of ten push-ups per 
minute. 

6. The pupil demonstrates his knowledge of the principle of simple 

harmonic motion by constructing a device which operates on this 
principle. 

Which of the preceding objeciives obviously contain a mixture of two or 
more behavioral components, namely, cognitive, affective and psychomotor? 

The utilization of domain taxonomy serves to simplify the process of specifi- 
cation of objectives— however, few behaviors are purely cognitive, affective or 
psychomotor. We are accurate if we speak of highly cognitive or affective or 
psychomotor. 

I 
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ANSWER SHEET-WORKING PAPER I 



1. A i 

2. A 

3. C J 

4. A ] 

5. P 

6. C 
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WORKING PAPER 11 

Compare the following set of objectives— a, b, f, and g with those in the orig- 
inal set. 

As a result of ^program, the pupils will: 

a. develop an understanding of mathematics as demonstrated by the 

construction of an original numeration system. j 

b. Develop an appreciation of music as demonstrated by his frequent 
discussions on music, his borrowing of numerous records, his use of 

leisure time to attend concerts and his participation in music activ- \ 
ities after school hours, 
♦f. become more self-directive as demonstrated by the number of times . 
he designs a course jf study for himself, formulates his own ob- * 
jectives in the context of his studies, carries out his studies by him- 
self, and identifies his errors and his accomplishments. j 
g. have faith in democracy as measured by The Academic Freedom i 
Survey. j 
In the above samples, the ambiguous terms were retained but the terms as \ 
demonstrated were followed by specific observable behaviors or instruments 
which served to indicate to some degree that the objective was attained. , 
In sum— whenever possible, overt or observable behaviors should be desig- ] 
nated. When this is not possible-when we deal with crucial cognitive and 
effective behaviors which are covert or not observa. -there is a need to | 
search for specific behavioral indicators or instruments which designate that J 
the objective was attained. 

For Reference Use •, ui ^ 

Overt behavior —any observable behavior which is readily n^'jasurable 

Covert behavior —any behavior which is not observable and requires ^ 

indirect measurement by utilizing behavioral mdi- ; 

cators or products of the process 

Examples: 

Overt behavior —the pupil swims (He doesn't write an essay on How ] 

I swim. " 

Covert behavior —The pupil analyzes given data by producing a written j 

report stating the relationships and interrelation- i 
ships found in the data. 

Illustrative "Tales" \ 
Dr. Glen Heathers "C;?Mrc/j"Tale 
Dr. Robert Magers "The Bartender in Africa" 

10 i 
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WORKING PAPER II (Cont.) 

For Reference Purposes: 

In constructing objectives, we attempt wherever possible to avoid terms such 
as: 

understands 

understands the meaning of 

really understands the meaning of 

appreciates 

knows 

enjoys 

grasps the significance of 

really grasps the significance of 

has faith in 

values 

investigates 

sees 

develops an insight 

becomes familiar with 

covers 

thinks 

learns 

realizes 

comprehends 

develops the concept of 

develops an interest in 

Whenever possible, we utilize such terms as: 

writes 



^ recites 

I discusses 

I I* states 

I 1^ cites examples of 

I lists 

i r illustrates 

J X describes 

'\ constructs 

I solves 
compares by listing 

I contrasts by stating 

I ■ * analyzes by stating or recording 

I I selects 

t * differentiates by listing 

I I explains in written or oral form 

I identifies 

names 



I I points to 

i I defines 



WORKING PAPER III 

Establishing Performance Levels 

Having described what it is we want the learner TO BE ABLE TO DO, wc 
can increase the ability of the objective to communicate by teUing the learner 
HOW WELL we want him to be able to do it. Wc accomplish this by describing 
the criterion of acceptable performance. 

Which of the following objectives indicate how the learner will be evaluated 
anrf describe at least the minimum level of acceptable performance? Utilize a 
check in the spaces designated. 



The learner will: 

I . develop a critical understanding of the operation of the Target 

Tracking Console. 

2. identify by name each of the controls located on the front of the 

Target Tracking Console and state the function of each control. 

3. demonstrate his skill in using a microscope. 

4 demonstrate his skill in using a microscope by viewing at least 3 

slides and constructing an accurate diagram of each. 

S. solve quadratic equations. 

6. solve at least 8 out of 10 quadratic equations. 

7. Given a meter with several scales and a range switch, the learner 

will identify the scale corresponding to each setting of the range 
switch. 

8. Given a linear algebraic equation with one unknown, the learner 

will solve for the unknown without the aid of references, tables 
or calculating devices. 



In sample items 1 -6, you will note that the even items all contained the min- 
imum level of acceptable performance. In sample items 7 and 8, the terminal 
behaviors not only contained the level of acceptable performance but were 
further defined by providing the giv^n^ or allowances or the restrictions under 
which the learner would operate. 
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CHAPTER 3-MEASUREMENT OF OBJECTIVES 

/ GENERAL CONSIDERATIONS 

Some of the techniques of research can be applied to evaluation. Research in 
the us'oal sense, however, is designed to produce new knowledge. Evaluation is 
more often the assessment of the effects of the application of the new knowledge. 
Evaluation has as its primary function the appraisal of a practical educational 
activity. Also, definitive research results sometimes take years to emerge. 

Thus, application of the usual experimental design often conflicts with the 
principle that evaluation should facilitate the continual improvement of a pro- 
gram. Innovative projects especially will want to alter their treatments during 
the process. Even waiting for the end-of-the-year evaluation data is often too 
late to make decisions about plans and procedures. 




To review the program as it evolves, monitoring must take place during the 
total situation. Data collection devices and techniques must be applied to the 
most crucial aspects of the project, analyzed periodically, and reported as often 
as needed. 

End-of-the-projcct measures of effectiveness may be needed. The nature of 
the project may be such that the end is the only time measurement can be made. 
In any case, it is essential to relate outcomes to objectives and make rational 
analysis of the results. 

13 
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Remember, behavioral objectives constitute a client-oriented approach to 
measurement, that is, from the standpoint of the consumer or user. What is the | 
student able to do? Thinking in these terms, the professionals in the education 
process are not particularly important. It's the student who counts. This leads - 
to a very valid way to approach evaluation. .i 

What has happened to little Johnny Jones? Can his learning climate be a more "f 
efficient one? 




For each objective in your project there must be at least one activity which 
could reasonably be expected to achieve the objective. For every objective and 
its related activity, one or more evaluative procedures are needed. 




What are the criteria for assessing the degree to which the objectives were 
achieved? 

One main focus is what all this has done to the target population. This area of 
adequacy is fundamentally conc^ med with administrative decisions* Did the 
program accomplish what was planned for? How do we know it works? What 
relationship does what happened before have to what was the final result? 

Objectives cannot be measured in isolation. They must be related to the total 
educational environment* Take a look around before proceeding. There may 
be previous studies which relate to the project. Appropriate instruments may 
be available. Is there money for additional staff? Can test material be purchased? 
How about consultant service? Above all, how much data are available from 
somebody else? 

As a result of the needs assessment, lots of information will be available from 
any given school system -test scores, attendance records, probability studies, 
ethnic analysis, poverty indices. If any of these data can be incorporated into 
the eval\ ation, it often makes the process a lot less difficult, less time consuming, 
and less expensive. 

The final report may be stronger if it contains an evaluation done by an out* 
sider— one who can perform an independent analysis and judge the project on 
its objectives merits. It is easy to be biased about your own work. Keep in mind 
that failures contain as much and perhaps more good practical information as 
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If an outside organization is going to help with the evaluation, it is wise to 
have them in on the planning and proposal stage. Then as objectives and pro- 
cedures are developed, theevaluator can prepare a realizable type of evaluation 
strategy. This avoids the problem of finding out at the end of the project that 
they have been pursuing a direction contrary to the needs of i he program. 



// TECHNIQUES 
It is essential to specify statistical techniques which are appropriate to the 
kind of data collected. If the achievement can be demonstrated by counting, 
don't use a complicated design. 
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m For example: 

I Which paperbacks in a library are getting the most mileage? Check on which 

ones arc worn out first. Which ones are stolen? 
I Is the Community Service Center having a beneficial effect? What are number 

of police referrals in the neighborhood? How many broken windows? Has the 
ft pattern changed? 

Often a correlation may be applicable. Is intelligence a factor in performance? 
f After some program of instruction, collect intelligence scores and performance 

1 scores. If, by some miracle, the highest intelligence matches the highest per- 

^ formance and so on down the distribution, the correlation is 1.0 (perfect cor- 

j relation as shown in left scatter diagram below). On the other hand, when none 

of the scores match, the correlation is .00 (no correlation as shown in right 
T scatter diagram). Somewhere in between are more realistic correlations that 

will give a numerical estimate of a relationship. 

I 
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Do student improvements result from the program? Measurements taken at 
the beginning can be contrasted with subsequent measures. If the difference 
scores compared to chance are significant, sound conclusions may be drawn. 
Ask yourself, however, was the treatment employed responsible? The only way 
to be sure would be to control all the other factors. By all means try to do this, 
but if not possible, don*t throw the data away— just temper your conclusions. 
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The experimental— control group method may be applicable. A special pro- 
gram is given one group of students and the results compared to the results from 
another group not participating. Some measure should be taken at the begin- 
ning to establish their quality. Teacher "/I "could tiy the new program. Teach- 
er "ff' would not. After a pre-determined teaching period, the classes are 
measured again and the results compared. If the experimental group has made 
the more substantial gains, the results are encouraging. How about possible 
teacher differences? Perhaps each group should be split in half and the teachers 
each try both techniques or the program might be repeated with the teachers 
playing opposite roles. 
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A sampling method may he of assistance. Who are the subjects? Are they not 
another factor that is varied? Do all of them need to be included in the mea- 
surement? Maybe conclusions and generalizations can be made from some parts 
of the whole. A sample or samples could be chosen by any method that gives 
each subject an equal opportunif ' to be in the group. A table of random num- 
bers or the old hat trick are acceptable devices. 




******** 



A pure random sample may not be desired. Differences within the total pop- 
ulation may be an important factor— ability, economic strata, training, etc. For 
instance, intelligence is an important aspect in reading. Therefore, the sample 
of pu A\s to be evaluated should reflect the range and proportions of intelligence 
within the total group. 

The project may have some anticipated outcomes that do not seem directly 
measurable. Here the evaluator might take an opinion poll of the staff and 
clientele involved to get the answers. One objective of an educational service 
center migot be to enable towns to cooperate. The people involved (parents, 
teachers, administrators) could give their opinions as to whether towns did 

19 



cooperate. If so, was this a result of this particular center? Has there been more 

cooperation since the center's advent? There may be no better way than to ask 

in some organized fashion— questionnaires or interviews. These techniques 

could also apply to the evaluation of an in-service program for teachers. To be ^ 

sure, the ultimate evaluation would be to identify measurable effects on the 

children working with those teachers, but right now it may be important to 

how the in-service program is received by the teacher-participants. 




********* 



Observational techniques may be particularly appropriate in connection with 
studies of young children. Observations of the subjects might be made over 
closed circuit T.V. and behavior samples recorded. For example, count the 
times the student leaves his seat during a twenty-minute reading period. After 
six weeks of therapy, observations are made again to see if he is settling down. 
Here, results are strengthened if precautions are taken to select specific aspects 
of behavior to be observed, to quantify the observations, and to train the ob- 
server in advance. 
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III INSTRUMENTS 
Some of the evaluation instruments needed may be commercially available. 
A perusal of tiie Mental Measurement Yearbook (Buros, Oscar K.— see bibli- 
ography) is a good place to start. Help may be available from another project or 
the schools and colleges in your area. Don't be afraid to ask. You're bound to 
find someone interested in measurement related to your project. An instrument 
may be found that is suitable in toto or one that can be modified to your 
needs. 

Remember, there is no 'Vig/i/" standardized test but there is probably one that 
is more applicable than others for your situation. Make sure whatever your 
instrumentation is, that is is acceptable for the given population. Most com- 
mercial tests are of the normative variety. Scores are normally distributed and 
the results place each student on a continuum. This may be just the information 
desired. On the other hand, where the concern is the individui.:ization of in- 
struction, other kinds of measures may be more appropriate. How does the 
individual's achievement correspond to some desired performance standard? 
How docs the quality of his current performance compare to that of his previous 
performance? Criterion-referenced tests of this type are being used for national 
and other assessment studies. Many of the actual items used are available. Com- 
mercial tests of this variety are also on the market. 

Afterasurvey of allthe available instruments, there is often a need for some- 
thing n^w. If the technical resources are available in conjunction with the proj- 
ect, elaborate and sophisticated instrumentation can be developed. If these re- 
sources are not available, the staff may have to develop their own. 
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One good way of getting teachers involved in a project is to let them help make 
the instruments. Homemade te^ts or questionnaires should be quickies. For 
the best response, make it short, sweet, yet cover the subject. 




Consider the ease of handling and sconng. Be sure that the new is compatible 
with the things you already have. If it doesn't make sense, leave it out. The 
instruments should be clean, clear, precise; take as little time and logistical 
manipulation as possible. 

No matter who maKes the instruments, be they test experts or project staff, 
two basic questions should be asked and, if possible, answered. 

The first and foremost question is how valid is it? Does it measure what we 
want to measure, all of what we want to measure, and nothing but what we 
want to measure? 

Suppose a test has been prepared to measure achievement in the use of the 
English language. Does it really measure such achievement? First, we must 
agree as to the skill, knowledge, and understanding that comprise correct and 
effective use of English. Then examine the test to see what skills, etc. it calls for. 
In other words, match test content to course cont *nt. The extent that our ob- 
jectives are represented in the test, it is valid. 
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Frequently we can correlate our instrument with some other criterion or 
measurement. The result of the English test might be compared to the rank 
individuals received from their English teacher, or better yet, to the results of a 
section of a valid standardized test concerned with English. 



English Test 
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Teacher Rank 



A good positive relationship would help establish the instrument's validity. 

The second question with respect to any measuring instrument is how reliable 
is it? Would it reproduce the same results if the same individuals were measured 
again? One way to fmd out is to do exactly that. Repeat the use of the instrument 
on a selected group of students and compare with the previous scores. Another 
♦rick is called the spli!-half method. This is a matter of comparing results based 
on one half of the test with the results of the other half. A strong positive cor- 
relation should be expected from either of these methods if the instrument is 
reliable. 
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IV CONCLUSIONS 
It is usually advisable to prepare a status or a time chart for the evaluation 
process. Who is going to do what and when? Who is going to select the instru- 
mentation? Who is going to administer the instruments? Who is going to analyze 
the data? Who is going to prepare the reports and dissemination? When do all 
these things take place? When are reports due? Check from time to time to see 
what is going on. 

The evaluation techniques mentioned here are but a few suggestions to 
stimulate your imagination and originality. In these electronic-oriented times, 
measurement docs not need to be confined to the usual pencil and paper tests. 
The sky's the limit. Any and all possible procedures are worth exploring to gain 
assurance that students are improving as the results of our efforts. 
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CHAPTER 4-DISSEMINATION OF EVALUATION 

I Introduction 

A consummated evaluation is valuable only to the extent that the results are 
known and used. At the receiving end, there are several groups who need and/ 
or want to know about a project's progress. At the sending end, the project 
managers are motivated and/ or obligated to convey progress information. 
The transmitter's purposes may range from just providing information to 
supporting recommendations and modifications he advocates. 




I 
I 
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Effective dissemination of evaluation information, however, is not quite as 
easy and standardized as the picture above suggests. In fact, this essential 
phase of the evaluation cycle requires just as much planning and care as the 
initial design and implementation phases. You must develop evaluation re- 
ports whose formats, contents, and emphases are appropriate and meaning- 
ful to your various audiences. 
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II Communication Audiences 

Project audiences often can be placed into four broad categories: local, 
state, professional, and non-professional. These categories are not mutually 
exclusive. 



Professional Non-professional 




While it is possible, under certain circumstances, for an individual to fit 
into two or more categories, we are mostly concerned with primary group 
placement for dissemination purposes. 

At the same time, it is important to illustrate the different audiences within 
each broad category: 

Local— Professional 

1. Project's professional staff 

2. Professional educators on project 
policy and program committees 

3. Professional educators employed by 
participating school district(s), i.e., 
superintendent of schools, school or 
program administrators, teachers, 
other specialists. 

local— Non-professional 

1. Project's non-professional staff 

2. Non-professional members on proj- 
ect policy and program committees 

3. Board of education of participating 
school district(s) 

4. Elected/appointed community offi- 
cials, e.g., mayors, town councils, 
boards of finance, political leaders 

5. Other members of the community, 
e.g., parents, special interest groups, 
taxpayers, etc. 
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Above all if you think in client centered terms— the pupil may be our most 
important audience. 




1 
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State— Professional 

1. State Department of Education, e.g., 
protect staff. Advisory Councils, 
Commissioner of Education 

2. State professional organization, e.g., 
CEA, Music Teachers Association, 
Connecticut Association of School 
Administrators, Non-Public School 
Associations, etc. 

State— Non-professional 

1 . State Board of Education 

2. State Legislature 

3. Other associations and societies 

4. Citizens in general 

It should be apparent in going through this list that these various audiences 
differ at least in knowledge, experience, interests, and expectations in relation 
to your project. Of course, there are other inherent differences, such as sex, age, 
ethnic background, socio-economic status, religious belief, political affiliation, 
which may or may not be relevant to any particular project. The point is that 
one reporting process and medium will not enable you to effectively commu* 
nicate your project's progress to all your deserving audiences. The key task is 
to communicate information so that it is as appropriate and meaningful as 
possible. 
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Ill Evaluation Reporting 

Despite the flexibility and variation stressed in the preceding section, 
there are common elements which should appear in any adequate evalua- 
tion report. Below are the minimal specifications: 

A. Description of Project and Evaluation Methodology 

1. Project objectives and program 

2. Statement of evaluative indicators 

3. Who and what was evaluated 

4. Evaluation techniques used 

5. Period of time covered by evaluation 

B. Presentation of Results 

Several of the following methods may be used, depending upon the 
nature of data collected and processed: 

1. Statistical 

2. Charts/graphs 

3. Anecdotal — observations, incidents 

4. Case study 

5. Testimony— opinions, ratings, etc. 

6. Criterion data— grades earned, 
commendations/ promotions, products, etc. 

C. Analysis and Conclusions 

Here the evaluator provides whatever analysis and interpretation 
is necessary to relate evaluation data to project objectives. If there are 
unexplained factors or extenuating circumstances, these should be 
identified. Finally, weighing all the information presented, the evalu- 
ator draws conclusions about the project in terms of effect, adequacy, 
efficiency, and process. These concepts can be depicted in the following 
way: 

1. Effect. Performance or effect cri . i'* measure the results of effort 
rather than the effort itself. This n .*;uires a clear statement of one*s 
objective. How much is accomplished relative to an immediate goal? 
Did any change occur? Was the change the one intended? 

Analogy: This can be compared to measuring how far a bird has flown 
instead of merely counting the flappings of his wings. 
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2. Adequacy. This criterion of success refers to the degree to which 
effective performance is adequate to the total amount of need. Thus, 
a program of intensive psychotherapy for a small group of mentally 
ill individuals may show highly effective results, but as a public health 
measure prove thoroughly inadequate to meet the problem of mental 
illness in an entire community. Adequacy is obviously a relative 
measure, depending upon how high one sets one's goals. 



Analogy: To continue the analogy made in reference to effort and effect, 
one would like to measure how far the bird has flown in terms of where he 
has to go. 




3. Efficiency. A positive answer to the question, "Does it work?" often 
gives rise to a follow-up question, '7^ there any better way to attain 
the same results?" Efficiency is concerned with the evaluation of alter- 
native paths or methods in terms of costs— in money, time, personnel, 
and public convenience. In a sense, it represents a ratio between 
effort and performance— output divided by input. 
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Analogy: Employing again the now familiar bird analogy, it is like asking 
the questions: "Could the bird have arrived at his destination more effi- 
ciently by some other means than flying the way he did?'' "Did he take ad- 
vantage of air currents; did he fly too high or not high enough?" 




4. Process. In the course of evaluating the success or failure of a 
program, a great deal can be learned about how and why a program 
works or does not work. An analysis of process can have both admin- 
istrative and scientific significance, particularly where the evaluation 
indicates that a program is not working as expected. Locating the 
cause of the failure may result in modifying the program so that it will 
work, instead of its being discarded as a complete failure. Making 
sense of the evaluative findings is the basic reason for adding a con- 
cern with process to the evaluation study. Otherwise, one is left with 
the descriptive results of the evaluation, but without any explanations. 

Analogy: To complete our analogy to the bird, it is as if we tried to learn 
something about the anatomy of the bird or the principles of flight as a 
further step toward understanding its effect, adequacy, and efficiency. 
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IV Reporting Requirements 

Referring again to our audience descriptions, the extent, depth, and 
emphases given to the elements above will vary in reports to different 
groups. Normally, professional educators will expect and deserve more 
detail on programmatic and evaluative techniques. Measurements spe- 
cialists will look for details on data collection and analysis, while concerned 
school administrators and laymen, such as board of education members, 
may be more interested in overall appraisal and recommendations to im- 
prove adequacy and efficiency. Pupils, parents, and citizens in general may 
primarily want to know whether the project is any improvement over what 
existed before. 

In other words, the evaluation reporter must use his own judgment and 
can be exercise options in much of his dissemination. However, since many 
projects operate under mandated policies and procedures, the evaluation 
report made to the granting agency must meet certain requirements. It 
must be submitted at a specified time, follow an established format, and 
contain prescribed information. 
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CHAPTER 5-CRITIQUE 
OF END-OF-YEAR EVALUATION 
REPORT 

Introduction 

As the conclusion of this manual, you are invited to critique a project evalua- 
tion effort. The following report on Project UPSURGE is a simulation, 
designed to incorporate the basic evaluation elements. Some aspects of the 
report are strictly fictitious. However, other aspects dealing with techniques 
used, data presentation and analysis are taken verbatim from actual project 
reports. In this sense, it is a composite of reality. 

Your task is to evaluate this evaluation report. What are its strengths? What 
aspects are rather vague and uncertain? What does the report suggpst about 
the processes of evaluation planning and implementation? 

Instructions 

This total exercise should require no more than one hour of your time. To 
do it, get off some place where it's quiet and where you can work without in- 
tcrruption. First, read the report through. Read critically, keeping in mind 
the ideas and desirable practices presented in this manual. Next develop your 
own answers to the questions which follow the report itself. Prior to filling in 
your answers, you will want to refer again to specific sections of the report, 
relating the information presented to the judgments made. Finally, you can 
get some feedback on your analysis. In the appendix are the summarized judg- 
ments of others who've done this exercise. (No fair looking ahead). 



Good luck! 




END-OF-YEAR EVALUATION REPORT 

Project Title: Project UPSURGE 

Submitted by: Fred Jones. Director 
Date: July 15. 1969 

INTRODUCTION 

Project UPSURGE (Unified Programs and Services for Unusual Regional 
and General Education) has a multi-faceted program serving 1 1 school dis- 
tricts. Its components include curriculum development, special services, cul- 
tural enrichment, and in-service teacher training. This report covers 
upsurge's second year of operation. 

THE PROJECT GOALS 

The general goal of UPSURGE is to meet important educational needs in 
the geographic area which it serves. More specifically, the project objectives 
are: 

1 . To develop critical science skills and attitudes in pupils 

2. To improve students' attitudes toward mathematics 

3. To assist schools in developing programs for children with learning dis- 
abilities 

4. To improve the problem-solving and perceptual capacities of perceptually 
handicapped pupils 

5. To foster the development of instrumental music skills in elementary 
pupils 

6. To develop children's appreciation and understanding of the arts 

7. To provide improved in-service training programs 

A CTIVITIES A ND RESULTS 
Project activities and results are grouped below into the four basic com- 
ponents. 

1. Curriculum Development 

Work was continued in improving science and mathematics materials and 
instruction. In three schools more manipulative materials and exercises 
were introduced into the math program. Since attitudes toward math in- 
fluences motivation and performance, an attitude scale was developed to 
cover five areas (see page 38). A total of 486 students marked the rat- 
ing scale in September and m June, providing a control group. Of these, 
278 took part in the modified instructional program, thus serving the 
experimental group. 

The Wilcoxon Matched-Pairs Signed-Ranks test which employs the di- 
rection of differences between first rating and second rating, and also the 
relative magnitude of the differences was used. 
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The results of the evaluation process are contained in the following tables 
and explanations. 

The Total Group 
Section 1 (the math course) 

N T Sign. 

Control group (208) 35 214.0 

Experimental group (278) 36 297.0 



Section 2 (the instructional process) 



N 
23 
24 



Sign. 



T 
113.5 
81.0 * 



Control group (208) 
Experimental group (278) 

♦Significant at the 0.05 level 
From the scale ratings of the experimental group in this section more stu- 
dents rated mathematics as important, 15% more students thought that 
they should have more tests, more students said that the teacher could give 
more individual help, and the other questions were stated about the same 
on both ratings, including the one about the way mathematics was taught. 



Section 3 (the text materials) 

Control group (208) 
Experimental group (278) 



N 
19 
20 



T 

86.0 
74.0 



Sign. 



Section 4 (the aids) 

Control group (208) 
Experimental group (278) 



N 
25 
25 



T 
102.0 
100.0 



Sign. 



Section 5 (the classroom) 

Control group (208) 
Experimental group (278) 

♦Significant at the 0.05 level 



N 
13 
13 



T 

43.0 
17.0 



Sign. 



From the scale ratings of the experimental group in this section group 
work was more favorable, less teacher assistance was requested, and ex- 
periments were less favorable. 

Using the entire group and this particular statistical test the balance of 
the differences had to be extreme to produce significant results as large 
positive results could cancel large negative results, or vice versa. For the 
total group significant results occurred in the sections which contained 
information on classroom environment and the instructional process for 
the experimental group, and no significant results occurred for the con- 
trol group; therefore, attitudes were aUered for the experimental group, 
as measured in the identified sections and the direction was positive. 
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New instructional units in science were also introduced into a number 
of area schools. In each case, brief pretests were developed and admin- 
istered at the beginning of the unit. At the conclusion of the unit an ap- 
propriate achievement test was used. On the pre-test in meteorology ninth 
grade students did poorly on all the questions except number three. Both 
students and teachers felt this was due to lack of exposure and laboratory 
work in these areas. On the pre-test in astronomy students in ihe upper 
intermediate grades did fairly well on most of the questions, with the 
exception of questions 4, 5, 6 and 7. The geology pre-test was prepared 
for advanced intermediate and secondary school students. The differ- 
entiation between rocks and minerals, conceptualization of geologic time, 
and ability to analyze a two-dimensional representation of land forms 
were found lacking in most of the students. 



I 
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2. Special Services 

A continuing effort of the project is to stimulate the development of pro- 
grams for children with learning disabilities. Several workshops for teach- 
ers and administrators from area schools have been conducted. In addi- 
tion, specialized consultants on the project staff have conferred with local 
school staffs on individual pupil cases. To determine the project's impact 
on local school systems a questionnaire was developed and sent to the 
superintendents of the 11 school districts. In response to the question 
concerning " What special education activities that the project offers are 
most essential?'' 88 items were listed. When the items were assigned to 
broad service categories, the distribution was as follows: 

Clinical Services 22% 

Consultation 47% 

Inservice 31% 



In response to the question "What special education activities that the 
project offers are least helpful?" 25 items were listed . When the items were 
assigned to br^ad service categories, the distribution was as follows: 

Clinical Services 36% 

Consultation 12% 

Inservice 98% 

In response to the question "What other special education activities do 
you wish the project organization to consider?'* 50 items were listed. When 
the items were assigned to broad service categories, the distribution was 
as follows: 

Clinical Services 40% 

Consultation 22% 

Inservice 22% 

A special program was instituted at Towaug Elementary School (East 
Overshoe) for perceptuajly handicapped pupils. 12 pupils on the basis of 
diagnostic testing were referred to a special class conducted by perceptual 
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specialists. This instruction began in October and continued through 
May. During the spring each child was reexamined with the same test 
battery which included the WISC, the Bender Gestalt, the Frostig De- 
velopmental Test of Visual Perception, the Illinois Test of Psychol- 
inguistic Abilities, and the Draw-a-Person Test. The test administration 
and interpretation was done by a well qualified clinical psychologist, (see 
comparative results and interpretations on page 42). In general,'6 
pupils showed marked improvement on both mental ability measures 
and perceptual tests. 4 other pupils showed variable gains, while the 2 
who shewed little or no improvement gave signs of interferring emotion- 
al difficulties. 



Cultural Enrichment 

Several project objectives deal with increaj^ing pupils' skills in and appre- 
ciation of the arts. A number of project activities were directed at t^ese 
objectives. For example, during the year a range of cultural programs 
were presented at area schools. These included: The Air Force Band, the 
Baker l.^arionettes, Oriental Dance Fantasy, the Short Pier Theater 
Company, and The Bach Rock. A post-performance questionnaire was 
completed by 210 high school students after seeing the Short Pier Theater 
Company's presentation. When asked to rate both the entertainment 
value and the educational value of this pro^jram, ratings of '^excellent" 
were given by 73<?? »nd 65%, respectively. 

Another successful activity is the instrumental music instruction being 
provided to many area pupils. A staff of 29 part-time instructors has given 
16,099 weekly half-hour lessons in fifteen different instruments to 580 
students. Students* progress is checked throughly and continually through 
music achievement testing and weekly skill progress ratings one by 
each instructor (see page 43), Findings with the Elementary Music 
Achievement Test indicate the following: 

1. First year pupils nade the following gains: 

pitch discrimination —17% Gain 

interval discrimination - 98% Gain 

meter discrimination - 27% Gain 

2. Second year pupils maae the following 
gains: 

pitch and rhythm discrimination - 92% above 

midpoint 

feeling for tonal color -50% above 

midpoint 

Instructors' progress ratings indicate that for the majority of students 
the development of performance skills was constant. Moreover, a group 
comprising about 15% of the total showed a very large degree of improve- 
ment. 
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4. Imervice Training 

Opportunities were provided to area teachers to participate in a variety 
of inservice training programs. Included were The Innovations Confer- 
ence, Workshop on Handicapped Pupils, Human Relations Training, 
Preparation A-V Materials, Use of Computers in Education, Arts 
in the Classroom, and Outdoor Education Seminar. The sensitivity 
aspect of Human Relations Training has created considerable interest. 
Seven (7) labs have been conducted with a total of 173 participants. Self- 
rating scales are completed at the beginning and end of the training by 
each participant. In addition, follow-up study is done when the partic- 
ipants are back on the jc '\ Our judgment is that the lab training is quite 
successful, although two sessions fell below expectations. 

Another popular training program is entitled Preparation of A-V Ma- 
terials. Its values are attested to in the completed questionnaires (see 

page 47). 



I CONCLUSIONS 

Project UPSURGE has continued to make progress in serving the educa- 
I tional needs of its area. It has offered new and more effective instructional pro- 

I grams in the sciences and mathematics. Student motivation and achievement 

in these fields have increased accordingly. The efforts of the special services 
I staff have stimulated greater awareness of and support for special education 

' in area schools. At the same time, we are redirecting project special services to 

better meet the desires of school superintendents. Of particular note is the 
I small, but important, demonstration of education for perceptually handi- 

^ capped children conducted in East Overshoe. The benefits of this program 

merit wider dissemination. 
I Cultural programs provided to area schools were considered to be of hign 

quality and were well received. At least one '*live** performance was experienced 
I during the year by every pupil K-12 in the area. In addition, almost 600 pupils 

I had the enriching benefit of instrumental music instruction on a regular basis. 

There is little doubt that these combined activities are reaching the objectives 
I which were set for them. 

I Finally, the attitudes and competence of area teachers have been enhanced 

by the range of inservice education which the project has sponsored. Participa- 
I tion in these programs has been high and many favorable comments have been 

I received. In the cc.ning year we plan to offer again those inservice programs 

reported herein and expect to develop several new workshops. 

I 
I 
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ATTITUDE SCALE USED TO MEASURE STUDENT INTEREST 



The purpose of this paper is to request your assistance in the construction of 
an attitude scale with respect to mathematics. You are asked to check the state- 
ments which express your true feelings about mathematics and those school 
elements related to mathematics. You do not have to sign your name to the 
paper; therefore, select the answer that you feel will best express your thoughts. 
Perhaps your answers will help to improve the instruction of mathematics once 
the teachers know of the likes and dislikes of the students. 



Section I 

I hate mathematics 

I am afraid that I will get the wrong answer 

I make low grades in my math courses 

I can't understand mathematics 

I avoid mathematics when I can 

I don't understand why I have to take mathematics 

1 can't get any help at home 

Math is my worst subject 

I can't stand verbal (word) problems 

I am very slow in working any kind of problem in math 

I dislike math 

I work the problems because I have to work them 

I wouldn't work some problems because they are hard 

» wouldn't try the problems if the teacher didn't check them 

I can work some verbal (word) problems 

I want to be able to work math problems 

Math is important 

Math is interesting 

I like math but not better than other subjects 

I don't dislike mathematics 

Verbal (word) problems are hard for me to do 

I work the problems in math because they are easy 

I work problems in math because they are useful 

I work the problems in math because they are fun 

I like mathematics 

The math problems take less time than other assignments 

I am rather fast in solving (finding the answer to) many kinds of pio 

math 

I like to do verbal (word) problems 

Math is my best subject 

I would like to spend more time working with mathematics 

I make above average grades in my math courses 
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I can explain some of my math problems to my parents 

I can see mathematics in use outside of school 

I usually get the right answers to my problems in mathematics— 

I enjoy doing problems in mathematics 

I rank math above all other subjects in my liking of subjects 

Section 2 

I do not like the way mathematics is taught 

I do not understand what my teacher is trying to say 

I get low grades in mathemat ics 

I would skip mathematics class if I could 

We never do anything different, just the same old thing 

I can't keep up with the rest of the class 

The teacher doesn't have time to give individual help _ 

Everyone must do the same problems during the same period— 

I have too many problems to do 

We have too much homework 

We have too many tests 

It takes too much time to do the assigned problems 

Mathematics is taught ok but I still have a lot of trouble with it.. 

My teacher says that math is important 

The mathematics class could be more interesting 

My teacher knows the mathematics ok 

I can see other ways in which the class could be made more in- 
teresting 

We have a lot of problems to do 

The teacher uses examples to help us work the problems 

The teacher makes the math class fun 

I think the math teacher is very good in mathematics 

I like the way mathematics is taught 

I would like to spend more time in math class 

The assignments do not take me very long to complete 

I think that we should have more tests 



Section 3 

I do not like the textbook that is used in my math class 

The book is too hard 

The book is too hard to read 

The problems in the book are too hard 

The book is old 

The book contains too many problems 

1 use the book in school 

I use the book at home 

I wish that we didn't have just one math book 

We work only from the book and have no worksheets 

1 don't see why we have to finish the book 

Some topics are covered too fast 

The book should have more explanation.-^ 

The mathematics book is ok 

I could learn the mathematics from the book if the teacher left.. 
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The book contains several examples that I can usually follow., 
I can read the book and understand what it is that I am to do.- 

The book is a new one 

I am supposed to be learning new math 

I think that this book is easy, 

I like the textbook that is used in my math class 



Section 4 

I don't think that the teacher should use any aids to help teach the math 
course 

I think that our textbook is enough to help me learn mathematics 

I do not like for the teacher to use an over-head projector in the math 
class. 



I like for the teacher to use the chalk board when he teaches math 

I don't like to use a lot of material in math class 

I don't like to see films in mathematics class 

I can't see using a lot of materials in mathematics class 

Some materials might be alright to use in a math class 

1 don't know what a model is 

A film can make math class interesting 

I sometimes use other materials in mathematics class 

I don't think that 1 would like to build things for a math class 

I might like to use materials if we had them 

My teacher uses materials like compasses and protractors in class to show 

something about math 

I am not impressed by the diagrams or materials used by the 

teacher 

I wish that math class had a lab 

1 like to see the teacher use other materials besides the chalk 

board 

I think that the use of models and diagrams in class make math class 

interesting 

I would like to be able to use the materials myself 

I would like to have a math lab in school 

I wish that my teacher would use more materials to make the math class 

interesting 

I wish that we had more materials to use. 



I would like to do experiments in the mathematics class 

I would like to read books concerning math or perso ns who use or write 

about mathematics 

1 think that the teacher should use any aio: to help teach the mathematics 

class 



ERIC 



40 



Section 5 

I don't like the room in which we have mathematics 

The math room does not have any equipment outside of a chalk board 

help learn mathematics 

We must always work in the same place and by ourselves 

We have too many persons in the math class.. 

I don't like to work in groups on the math assignments. 

I wish that we could move our chairs around and work together 

I think that we could make our classroom better for learning 

mathematics 

I don't care to work by myself doing a math experiment ^ — 

I wish that the teacher would let us learn by ourselves sometimes 

Experiments in the math class might be fun 

I think that I would like to work in groups while doing 

assignments 

I think that the room is alright in which we have math class- 

I would like to change the room to help me learn mathematics 

better 
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Subj. 


1st 

Tested 


Last 
Tested 


Verbal 
l.Q. 
1st 


Verbal 
IQ. 
Last 


Perf. 
IQ. 
1st 


Perf. 
l.Q. 
2nd 


Full 
LQ. 
1st 


Full 

l.Q, Bender 
2nd 


Frosting 


ITPADrawing 


Observation 




















much 


much 


marked 


1 


10/6/66 


4/22/68 


119 


128 


99 


no 


110 


121 imp. 


imp. 


— imp. 


improvemeni 




















much 


much 


markc^^^ 
improvement 


2 


3/13/67 


4/25/68 


130 


135 


121 


127 


128 


134 imp. 


imp. 


— imp. 


















much 






improvement 


3 


3/23/67 


2/29/68 


99 


92 


83 


99 


91 


95 imp. 




imp. 


perf. only 


4 


5/ /67 


5/23/68 


99 


95 


78 


80 


88 


87 - 


slight 
imp. 




lack of 
motivation 


















much 




much 


marked 


5 


3/6/67 


6/ 13/68 


91 


99 


75 


93 


82 


96 imp. 


imp. 


— imp. 


improvement 
























slight 


6 


5/1 '67 


5/23/68 


85 


87 


76 


79 


79 


82 - 


imp. 


~ imp. 


improvement 


7 


1/30/ 67 


3/7/67 


121 


116 


117 


128 


12! 


not 
124 imp. 




— emotional 


improvement 
perf. only 


8 


4/6'67 


4 '25/68 


125 


135 


129 


138 


130 


slight 
140 imp. 




— imp. 


marked 
improvement 




















not 




complications 


9 


?/;/66 


4-4/68 


109 


108 


72 


69 


91 


88 imp. 


imp 


— imp. 


evident 
























improved 


10 


2, 10/66 


4 11,68 


91 


84 


101 


107 


96 


94 imp. 




imp imp. 


per. only 


11 


3/27^67 


3' 28 68 


97 


99 


69 


83 


83 


slight 
9! itup. 




— imp. 


marked 
improvement 
























marked 


12 


6/ 19 67 


6/ 10/68 


89 


99 


93 


111 


90 


105 


imp. 


imp. — - 


improvement 
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EVALUATION FORMS FOR INSTRUMENTAL LESSONS 

STRING FAMILY 



Name of Instructor 

Instrument being taught. 

Name of Student— — - 

Address 

Age School ^Grade 

Boy Girl 

Previous Instrumental Experience 

Attendance (P-present; A-absent) 



WEEK 

1. Ability to hold instrument and bow 
correctly; to draw bow with good wrist 
motion. 

2. Ability to play with good tone. 

3. Ability to recall learned notes on each 
string. 

4. Ability to sight read. 

5. Ability to play lesson assignments 
fluently. 

6. Ability to play with proper rhythm. 
1. Ability to play with accurate pitch. 
8. Ability to understand signs. 

Please use the following scale in marking: 
1 -Excellent; 2-Good; 3-Fair; 4-Poor 



43 



EVALUATION FORMS FOR INSTRUMENTAL LESSONS 
WOODWIND AND BRASS 



Name of Instructor 

Instrument being taught 

Name of Student 

Address 

Age School ^Grade 

Boy Girl— 

Previous Instrumental Experience 

Attendance (P-present; A-absent) 



WEEK 



1 . Ability to hold instrument properly. 

2. Ability to play with good tone. 

3. Ability to play in tune. 

4. Ability to tongue. 

5. Ability to recall fingering for all 
learned notes. 

6. Ability to play with mouthpiece used 
correctly. 

7. Ability to play lesson assignments 
accurately. 

8. Ability to sight read. 

9. Ability to play in rhythm. 

10. Abilityto read and understand 



Please use the following scale in marking: 
1 -Excellent; 2-Good; 3-Fair; 4-Poor 
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EVALUATION FORMS FOR INSTRUMENTAL LESSONS 

PIANO-DRUMS 



Name of Instructor^ 

Instrument being taught 

Name of Student ^ , 

Address . 

Age School ^Grade 

Previous Instrumental Experience 

Attendance (P-present; A-absent) 



WEEK 

1. Ability to play with proper posture 

and hand position. 

2. Ability to read learned notes. 

3. Ability to play rhythmically. 

4. Ability to sight read. 

5. Ability to play lesson assignment. 

6. Ability to recall and understand 

signs. 

7. Ability to use two hands alternately 
and then together. 



Please use the following scale for marking: 
1 -Excellent; 2-Good; 3-Fair; 4-Poor 



45 



INSTRUCTORS* SKILL PROGRESS RATING FORMS 
October 1968— May 1969 



Considerable 
Improvement 



Improvement Improvement Regression 



Constant No 



October 1968— November 1968 

December 1968— January 1969 

February 1969— March 1969 

April 1969— May 1969 
YEARLY AVERAGE 



18% 

18% 

13% 

14.2% 
15.8% 



67.5% 8.5% 6% 

46.0% 30.0% 6% 

50.0% 32.0% 5% 

68.0% 16.2% 1.6% 

57.9% 21.7% 4.6% 



Explanations: 

A weighted rating scale from 1 to 4 was used. Weight 1 means excellent; 
weight 2 means good; weight 3 means fair; and weight 4 means poor. 

Considerable Improvement— Indicates marked tendency toward higher 
ratings, for example, a frequent occurrence of ratings of a weight of 3 early 
in the two month period is replaced by ratings of a weight of 2 and a weight of 

1 in the two month period. 

Constant Improvement — Indicates continuous high ratings of a weight of 

2 and a weight of 1 for the two month period (with an occasional rating of a 
weight of 3 or a weight of 4.) 

No Improvement — Indicates no improvement in ratings for the two month 
period, for example, the student began the period with a rating of a weight of 

3 and/ or a weight of 4 and continued to receive a rating of a weight of 3 and/ 
or a weight of 4. 

Regression— lnd\c2iXcs a marked tendency toward lower ratings, for ex- 
ample, a rating of a weight of 2 and/ or a weight of 1 early in the two month 
period is replaced by a rating of a weight of 3 and /or a weight of 4 in the latter 
part of the two month period. 

Table 6 shows that the largest proportion of students showing constant im* 
proveihcnt, the same at the end of the year as in the beginning of the year in 
the level of skill, as measured by a weighted rating. That is, 67.5% received 
continuous high ratings of a weight of 2 and a weight of 1 for the first two 
month period. In the final two month period, 68% were in this category. 

The "no improvement" category ranged tremendously over the time span. 
The 30% of students showing ''no improvement" in December — January and 
February— March (32%) appears as a direct resuh of absence peaks resuhing 
from bad weather conditions and a flu epidemic. 
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IN-SERVICE WORKSHOP 



EVALUATION FORM 



Did objectives of workshop meet with your objectives? Yes-^ — No. 

3rd & last 

Which session did you feel was of most value? 



Have you been able to use any of the information and skills in your teaching? 

Yes NoJ^>'^' 



Would you suggest any other topics for a workshop?- 



Comments about length of each session and /or number of sessions: 



Any further comments: 

Missed session on transparencies, would like to learn process. 
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IN-SERVICE WORKSHOP 



erJc 



EVALUATION FORM 



Did objectives ofworkshop meet with your objectives? Yes— — No- 

3.4 &5 

Which sessions did you feel was of most value? 



Have you been able to use any of the information and skills in your teaching? 

Yes_^ No 



Would you suggest any other topics for a workshop?- 



Comments about length of each session and /or number of sessions: 

Perhaps a doubling or duplication of each session ( video-tape)f or people 
who missed a session. 

Any further comments: 



4% 



IN-SERVICE WORKSHOP 



EVALUATION FORM 

Did objectives of workshop meet with your objectives? Yes-^ — No 

Transparencies 

Which session did you feel was of most value? 

Have you been able to use any of the information and skills in your teaching? 
Yes-^^ No 

Would you suggest any other topics for a workshop? 

A more in depth approach to research materials that may be used in a Soc. 

Stud. Curricula- Where to go- Who to write to- What sources are available- 

Cataloges of free & paid films, etc.. etc. 
Comments about length of each session and / or number of sessions: 

Time— Earlier in the week would be better. Two hours at end of week is rather 

hard to accept personally. 



Any further comments: 

Would it be possible to have such a .session for in-service workshops for 

any particular school district— possibly paid for by the school requesting your 

services. 
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IN-SERVICE WORKSHOP 



EVALUATION FORM 



Did objectives of workshop meet with your objectives? Yes- 



No- 



Which session did you feel was of most value? 

session on lamination and final session on LV, 
Have you been able to use any of the information and skills in your teaching? 

Yp c ^ No 



Would you suggest any other topics for a workshop?- 



Film making— operating 8 and /or 16 mm. cameras 



Comments about length of each session and / or numb: ^ i^f sessions: 

Adequate — I feel one session on Mechanics was too much. 
More time on V, 



Any fui ther comments: 



SERlCf 
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IN-SERVICE WORKSHOP 



EVALUATION FORM 



Did objectives ofworkshop meet with your objectives? Yes_l£ No 

Dry mount &. T. V. Taping 

Which session did you feel was of most value? ^ 



Have you been able to use any of the information and skills in your teaching? 

Yes-^^^ No 

Dry mounting so far 

Would you suggest any other topic^ for a workshop? 

on-resources, ways & means of finding materials for use in all curricula 
areas— 

Comments about length of each session and / or number of sessions: 
Number of sessions— like more! 

Length— suggest one hour— 2 hrs. is a bit much for me on a Thursday. 
Anyfurthercomments: 

Yes! Hov/s about ihe future possibility of this being an extended course 
that might be used as credits towards a degree through WESCONN. or 
U.B.,etc. 
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IN-SERVICE WORKSHOP 



EVALUATION FORM 



Did objectives of workshop meet with your objectives? Yc$^ — No — 

Making transparencies. 

Which session did you feel was of most value? 



Have you been able to use any of the information and skills in your teaching? 

Vp c No 



Would you suggest any other topics for a workshop?- 



Making slides andfilmstrips to run with Controlled Reader. 
Comments about length of each session and/ or number of sessions: 

Follow up lessons for individ. preferences might be helpful as there seemed 
to be so much to cover in such a short time. 



Any further comments: 

Missed Dry Mount session would be interested in learning this technique. 
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IN-SERVICE WORKSHOP 



EVALUATION FORM 



Did objectivesof workshop meet with your objectives? Yes^-^ No. 



Which session did you feel was of most value?- 



Transparencies for overhead projector 



Have you been able to use any of the information and skills in your teaching? 



Yes- 



No- 



Would you suggest any other topics for a workshop? 



No 



Comments about length of each session and/ or number of sessions: 
Can U see anything wrong with sessions as arranged.. 



I 

1 
1 



Any further comments: 



Because I was not familiar with constantly occurring common terminology, 
it took me longer to absorb the point and at times, I didn '/ absorb the point. 
This certainly could he more the result of my being so new to the profession 
rather than an oversight on your part. Wish you had distributed a general 
information sheet before each lesson pertaining to equipments— prices, 
whereabouts etc. 
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IN-SERVICE WORKSHOP 



EVALUATION FORM 



Partly 

Did objectives of workshop meet with your objectives? Yes No 

Dri-mounting'lamxnation 
Which session did you feel was of most value? — _ 



Have you been able to use any of the information and skills in your teaching? 

Y^s ^ No 



Would you suggest any other topics for a workshop?. 



More use of catalogs and information lists of available materials. 

Comments about length of each session and/or number of sessions: 

Two hours seems a little lonf with the amount of available equipment 
to use for practice. A Iso the latter part of the week is a bad time to meet. 



Any further comments: 

Workshops held at individual school on equipment which is there to use 
would prove of more lasting value. In-service credit would be an added 
incentive. 
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CRITIQUE QUESTIONS 



I Project Objectives 
1 ) Areall objectives stated as behavioral outcomes? 



(Yes/No) 

2) Are some objectives stated as behavioral outcomes? 

Which?^ 

(Write in numbers) 

3) Which ^^objectives' are really project activities (inputs rather than outcomes)? 



(Write in numbers) 

11 Evaluation rocedures 

l)Are the evaluation populations identified as to number, project involvement, 
and other important characteristics? 



(Yes/ No/in some cases) 
2) For how many program elements are no evaluation data reported? 



3) Which of the evaluation techniques used are clearly defined for the reader? 



Which are not? 



55 



4) Which evaluation techniques used provide adequate and appropriate evidence 
of achievement of objectives? 



ERIC 



1 1 1 Results and Conclusions 

I) Which results of the evaluation are objective and clear-cut? 



Which results are not and why? 



2) Which of the conclusions drawn are basically valid? 



Which are not and why? 
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APPENDIX 



J CRITIQUE OF EVALUATION REPORT 

I Project Objectives 

r 1) Are all objectives statedas behavioral outcomes? No 



Yes 

2) Are some objectives stated as behavioral outcomes? 

Numbers 1, 2, 4, 5, 6 



Which? 

(even these do not include full identification of the evaluation population, techniques 
to be used, and desired level of performance) 



3) Which "objectives'' 'dvc really project activities(input rather than outcomes? 

Numbers 3, 7, 8 
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II Evaluation Procedures 

I) Are the evaluation populations identified as to number, project involvement, and 
other important characteristics? 



some ':ases 



2) For how many program elements are no evaluation data reported? 
13 — no post'test data on 



three science units, no data 



on four cultural programs mentioned. 



no data on six inservice training programs 



3) Which of the evaluation techniques used are clearly defined for the reader? 
math attitude scale, testing , 



used with perceptually handicapped pupils. 



measures of musical achievement 



. , ^ pre-tests and 
Which are not? L 



post-tests in science, special 



services questionnaire, cultural 



program questionnaire, rating scales in 



sensitivity traming 
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4) Which evaluation techniques used provide adequate -and appropriate evidence of 
achie\enient of objectives? 

Sections of math attitude scale deal with ranee of attitudes . — 

but all based on self-report: Music instructors' ratines are 



frequent and behaviorallv defined but could reflect personal 
bias: Pre and post testing of oerceptuallv handicapped p upils 
provides for relevant and comparable measures. 

III Results and Conclusions 

I) Which results of the evaluation are objective and clear-cut? 

The data on the perceptually handicapped program; 



The results on math attitude improveme nt and musical achievement 

are encouraging although thev are based o n single evaluation 

a pproaches which can be heavilv influe nced bv the person 

responding. 



cut because of uneven evaluation coverage, undefin ed techniques 
used, and inadequate collection and p resentation of data. 



2) Which of the conclusions drawn are basically valid? 

The benefits of the perceptually handicap ped program do merit 



wider dissemination, considering the marked imp rovement in 50% 

of the cases: there has be^ " prn^»,rfts»; in the development of 

musical skills in the majo rity nf nupils participating. 



Which are not and why? 

There is nn evidence that ••xiudent moti vation and achievement in 

(math and ^riPnre ) have increased accordin gly"; one would be hard 

pressed to know how to redi rect the special services program "to — 

better meet the desires of s chool superintendents" on the basis 

of this report: the evaluatio n nf the in-school cultural programs 

is almost entirely absent i n this report but •'there is little 

doubt that ft his activit v is } rpachine the objectives which were 

set for (it f. Finally, the evidence pres ented provides no founda- 



Which results are not and why? | 
None of the other resultscan he considered objective and clear- 



Jinn for the assertion that ''the altitudes and competence of area 

teachers have be i-" ^"^^^^^^^ b v the ranee of inservice education which _ 
the project has sponsored. " 



O All 
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-Bureau of Pupil Personnel 
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Services 
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